THE DATA CRUNCH 


“WITHOUT VALID AND RELIABLE DATA 

we are simply flying by the seat of our pants, and no serious, 
objective-minded policy wonk is going to pay any attention to our 
pleas for support." — Professor Richard Lynch. 


The value of GTE is best demonstrated 
by comprehensive community impact 
measures in addition to the highly valued 
educational attainment measures we are 
accustomed to.” 

Using Data Wisely 

Lynch advises that, “Quantifiable data, 
important as it may be, should never 
replace the wisdom and judgment of 
teachers, other practitioners and parents. 
Good teachers know students well and 
know how students learn; their collec- 
tive wisdom can and should be used in 
powerful ways to improve and strengthen 
CTE.” 

Lynch also believes that CTE needs 
to take the lead to expand the conversa- 
tions with policy groups, legislators and 
education reform advocates about what it 
means to achieve. “Student achievement 
is far, far more than a score on a norm- 
or even criterion-referenced standardized 
test,” he adds. “Further policy edicts just 
must include the measurement of student 
achievement through successful assess- 
ment of student work on age-appropriate, 
complex projects, occupational certifi- 
cations earned, state licenses awarded, 
and other demonstrable measures of 
competence. Other factors might also 
include attendance, graduation, transition 
into employment and/or postsecondary 
education, scores on standardized tests 
that measure real-world skills, assessment 
of soft skills, and student organization 
competitions. Overall, I think it only just 
and fair to showcase student achieve- 


ment through a portfolio of student work 
and accomplishments, and yes, this can 
include scores on standardized tests of 
academic achievement.” 

Over the years, Lynch has noted that 
many fine CTE administrators and 
teachers are cautious or defensive about 
disseminating any data, especially of the 
accountability type that state and federal 
agencies have called for, and he thinks 
that this may be the cause of data tend- 
ing to languish in those repositories as 
noted by Wayman. “I think such fear or 
hesitancy of releasing data is unfortunate, 
as it is pretty difficult to improve if the 
data aren’t known or transparent,” he 
adds. Instead he would encourage those 
who have concerns about disseminating 
such data to try to better understand what 
is being asked of the data and, hopefully, 
be involved in setting the objectives of the 
data collection. 

“Conceptually, it is pretty simple,” 
Lynch explains. “What is it you need to 
know? How will you come to know it? 
How will you know what you got? And 
then use the data accordingly; that is, to 
either advocate for more and improved 
CTE and/or to make the modifications 
that are warranted.” 

He sums it up this way: “Without valid 
and reliable data — including that trian- 
gulated through qualitative measures and 
the wisdom of teachers — we are simply 
flying by the seat of our pants, and no 
serious, objective-minded policy wonk is 
going to pay any attention to our pleas for 
support.” D 


Exploring Sources of Data \ 

For more information about harnessing \ 
the power of data to make a case for the \ 
benefits of career and technical education, 
here are some sources to explore. 

Association for Career and Technical 
Education (www.acteonline.org) 

Information and Research (www.acteonline. 
org/content.aspx?id= 206 ) 

National Research Center for Career 
and Technical Education 

www.nrccte.org 

Center for Occupational Research 
and Development 

www.cord.org 

National Dropout Prevention Center 

www.dropoutprevention.org 

The Bill and Melinda Gates Foundation 

www.gatesfoundation.org 

Data Quality Campaign 

www.dataqualitycampaign.org 

Southern Regional Education Board 

www.sreb.org 

About the Schools 

To learn more about the schools featured 
in this story, visit their Web sites. 

Gateway Technical College 

www.gtc.edu 

Metro Technology Centers 

www. m etrotec h . o rg 

Francis Tuttle Technology Center 

www.francistuttle.com 


Susan Reese 


is a Techniques contributing writer. She can be contacted at 
susan@printmanagementinc.com. 

Interested in exploring this topic 
further? Discuss it with your 
colleagues on the ACTE forums at www. 
acteonline.org/forum.aspx. 
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ft Perkins challenge: 

Assessing Technical skills in CTE 



BY JAMES R. STONE III 

C urrent federal law requires states to develop 
performance measures and data collection 
systems for the four required core indicators, 
including technical skill attainment for sec- 
ondary and postsecondary levels. These measures must be 
reliable and valid. Perkins IV requires states to use state- 
established, industry-validated career and technical skill 
standards; but few of these exist and states have largely been 
compelled to rely on career and technical education (GTE) 
course completion, grade point averages (GPA) in CTE 
courses, or administrative data. 

Challenges 

The first challenge posed by the Perkins requirement is 
defining a technical skill. How do we parse technical skills 
from non-technical skills? The most common approach 
relates technical skills to skill standards which are “perfor- 
mance objectives and competencies required by a specific 
occupation, as specified by experts within that particular in- 
dustry” (Center for Remediation Design, 1991), and “perfor- 
mance specifications that identify the knowledge, skills and 
abilities an individual needs to succeed in the workplace.” 
According to the National Skill Standards Board, skill stan- 
dards “. . .consist of two components: (1) a description of the 
responsibilities needed for competent performance, and (2) 
a description of knowledge and skills necessary to carry out 
these responsibilities” (2000). 

A second challenge is measurement. True performance 
assessments, sometimes known as authentic assessment, 
require a student to perform. Performing a task, the process 
of doing, rather than selecting an answer from a ready-made 
list (circle “D” for “none of the below”) is a hallmark of per- 
formance assessments. The U.S. Congress Office of Tech- 
nology Assessment defines performance assessment as “any 
form of testing that requires a student to create an answer or 
a product that demonstrates his or her knowledge or skills.” 
This requires experienced raters who judge the quality of the 
student’s performance against established criteria; consider 
driver certification tests or Olympic events. Closer to home, 
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one need only look to the competitive 
events sponsored by CTSOs to see how 
this might be done. DECA events, for 
example, require both a test of factual 
knowledge and a performance. 

An authentic performance assessment 
relies on experts among whom there is 
high inter-rater reliability. Those who 
have experience with CTSO competitive 
event judging know the inherent difficulty 
in this process. As well, a commentary 
in a recent issue of Education Week points 
out the manifold problems with scoring 
of open-ended questions on the National 
Assessment of Educational Progress that 
ought to give one pause before advocat- 
ing such a system for CTE technical skill 
assessment. Another factor to consider 
in this process is logistics and attendant 
costs. Imagine if the SkillsUSA or DECA 
competitive events were adopted as the 
model to meet the Perkins accountability 
requirement. Also, there is the need to 
establish these measures as being both 
valid and reliable; this can be an extraor- 
dinarily expensive proposition. 

Another challenge is timing. When 
is a student assessed? Presently, among 
states that do assess, they do so at the end 
of a program. The current evidence on 
high school student enrollments sug- 
gests that fewer students are complet- 
ing programs than previously. Should 
assessments be done at the end of course 
to ensure we capture sufficient data to 
improve programs? Do end-of-course 
assessments in these high schools truly 
reflect technical skill attainment? Also, 
Perkins IV now requires that all states 
have programs of study/career pathways 
for all CTE students; connecting second- 
ary and postsecondary CTE and assess- 
ing and reporting progress at the end of 
completing these pathways may be more 
relevant for determining progress and 
success in CTE. 

An attendant issue is measurement to 
what end? That is, what use will be made 
of the data generated by the technical 
skill assessment? The National Research 


Center for Career and Technical Educa- 
tion convened a small panel three years 
ago to discuss this issue. The panel in- 
cluded industry representatives, experts in 
technical skill assessment, as well as lead- 
ers from the CTE community. Among the 
results of this discussion was a rank list of 
important assessment functions: 

1 . Provide data for local program 
improvement. (A diagnostic 
function.) 

2. Provide a signal to the labor market 
or higher education. (A certification 
function.) 

3. Provide data that can be aggregated 
within a state and among states. 

(An accountability function.) 

Industry-Driven Models 

Assessing technical skills is not a new re- 
quirement; over the past decades at least 
three models have emerged, with a fourth 
under serious consideration. Industry- 
driven models have long been a part of 
CTE. The auto industry, for instance, 
has developed a partnership with the Na- 
tional Automotive Technicians Education 
Foundation and a nonprofit organization 
Automotive Youth Educational Systems 
(AYES) that has established a set of cur- 
ricular and programmatic standards de- 
signed to deliver high-quality, advanced 
skill training that is closely linked with 
workforce needs. Graduates of AYES- 
certihed programs are eligible to be certi- 
fied in brakes, steering and suspension, 
electrical systems and power train. 

In the IT industry, industry leaders 
have created certification courses in net- 
working and program language. Notable 
among these are Cisco, Novell, and 
A-Plus. Finally, the National Health Care 
Curriculum Consortium has partnered 
with the National Occupational Compe- 
tency Testing Institute (NOCTI) to create 
a certification exam used in 35 states to 
assess knowledge critical to a family of 
health care occupations. The process 
requires the use of post-program assess- 


ments that provide either a certificate 
of proficiency or a certificate of mastery 
targeted at broad understanding of the 
health care industry and related content 
area. These are in addition to the widely 
recognized certified nursing assistant, li- 
censed practical nurse or registered nurse 
certifications. These industry certifica- 
tions provide a very clear signal to the 
labor market about a candidate’s employ- 
ability. Aside from industry produced as- 
sessments, third party organizations like 
NOCTI also provide industry-validated 
assessments. 

State Models 

Massachusetts is an example of a state 
model in which employability certification 
programs are developed in cooperation 
with employers. The state has identified 
nine competencies that form the founda- 
tion for its work-based learning plan. Suc- 
cessful attainment of these competencies 
meets a second kind of certification — em- 
ployability. (The first certification is the 
academic Massachusetts Comprehensive 
Assessment System.) A third awaits those 
who successfully complete CTE programs 
in areas where the state has partnered 
with industry to develop industry-specific 
certifications. 

In addition, states like Utah are devel- 
oping end-of-course exams that presum- 
ably assess skills as well as knowledge. 
Success in applying these state models for 
measuring technical skill attainment can 
simultaneously meet the U.S. Department 
of Education’s twin conditions of third- 
party assessment as well as validity and 
reliability. However, if the requirement to 
increase student and program coverage is 
also in place, state models to implement 
technical skill assessment structures can 
be cost prohibitive. 

The CTSO Model 

As noted earlier, each of the eight 
CTSOs include industry-validated 
competitive events in their portfolio of 
activities. Using cognitive tests of related 
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Figure 1: Skills Pyramid 


Figure 2: Skills Pyramid 



content knowledge and performance 
activities that are industry judged and 
validated, the CTSO model presents 
another model for technical skill assess- 
ment. SkillsUSA is currently developing 
dozens of industry-based skill assessments. 
In general, where neither industry- driven 
models nor state models are feasible, states 
are increasingly turning to state-validated 
but locally applied assessment systems, 
which generally could be hybrids of the 
two models discussed above. Addition- 
ally, states have in place proxy measures 
for technical skill attainment such as the 
use of conventional academic success 
measures such as GPA, course completion 
and administrative data. While the latter 
methods of assessment are not commonly 
preferred under Perkins IV, many states 
do have them in place to measure techni- 
cal skill attainment and have included 
them in their state Perkins plans argu- 
ing that these methods are as valid and 
reliable as the third-party technical skill 
assessments preferred under Perkins IV. 

Challenges Ahead 

There is a new effort that focuses on 
the 16 career clusters and the 81 career 
pathways. The challenges of definition 
and measurement are perhaps more pro- 
nounced at this level. While measuring a 
health cluster student’s ability to “effec- 


tively communicate diagnostic informa- 
tion using written communication” may 
seem relatively straightforward, “employ- 
ing occupational safety techniques” poses 
greater measurement challenges. Both are 
identified as technical skills in the current 
career clusters framework. One way to 
think about skill assessment is to envision 
a pyramid that combines a mix of aca- 
demic knowledge and technical skills that 
changes as one moves from clusters to 
pathways to programs and on to specific 
occupational training (see figure 1). Using 
the health cluster as an example, one can 
envision a different kind of assessment 
for a cluster than one might create for a 
program (see figure 2). It is also likely that 
different constituencies (e.g., higher educa- 
tion, state government, teachers, industry 
and students) might attend to different 
assessments linked to different points on 
the pyramid. Many industries — for ex- 
ample health, auto or IT might focus on 
program or occupation-specific education 
while education agencies might focus on 
clusters or pathways. 

Given all of the above, basic and fun- 
damental questions remain: what is the 
most important focus for high school and 
college CTE technical skill assessments? 
Should we focus on diagnostics, signal- 
ing or accountability? Is it possible to 
do all three with one assessment? These 


are important questions confronting the 
held as CTE struggles to identify the 
best approach that benefits our students, 
their (future) employers, and the several 
educational entities that have an interest 
in these questions. D 
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